Developing And Integrating A Web-based Quiz Into The Curriculum

Angela Carbone Department of Computer Science
Monash University, Australia
Angela.Carbone@cs.monash.edu.au

Peter Schendzielorz Department of Computer Science
Monash University, Australia
cadal@ozemail.com-au

Abstract: In 1996 the Department of Computer Science, Monash University, implemented a First Year Advanced Students' Project Scheme aimed at extending and stimulating its best first year students. The aim of the scheme was to give students the opportunity to work on a project that best suited their needs and captured their interest One of the projects which became known as CADAL Quiz, (Computer Aided Dynamic Assessment & Learning Quiz), involved designing and implementing a World Wide Web based multiple choice quiz generator and assessment tool.

Unexpectedly, at the time, there were several academics wishing to move away from the traditional mode of educational assessment and move towards interactive, computerized assessment. As a result CADAL Quiz was incorporated into the First Year Computer Programming unit and utilized by lecturers, tutors and students.

This paper reports on a pilot project for developing and integrating CADAL Quiz into the curriculum. It highlights the unique quiz features, and its use by students and staff. The paper describes how the quiz was incorporated into the First Year Computer Programming unit and presents a conduit of attitudes useful to those who are planning to use the Web as a resource for educational assessment.

1. Introduction

With the onset of the Internet, in particular the World Wide Web (WWW), it has been increasingly popular to move away from the traditional mode of education and move towards a more interactive, computerised system. [Godfrey, 1996], [Conway, 1993], [Conway, 1994]. Such a move was simplified by the work of a devoted student who designed and implemented CADAL Quiz as part of the 1996 Computer Science, First Year Advanced Students' Project Scheme [Carbone, 1996), [Carbone, 19961.

CADAL Quiz is a multiple choice quiz generator and assessment tool that utilises the WWW. In 1997 CADAL Quiz was incorporated into the curriculum by first year lecturers. This change from paper based assessment to WWW based assessment is described in this paper as well as the various features of the CADAL Quiz package.

2. Design and Description of CADAL Quiz

CADAL Quiz, as used by the First Year Computer Programming unit, was designed with a number of goals in mind, including:

1. Providing shared ownership of assessment questions by tutors and lecturers. In the past

lecturers took the sole responsibility of setting assessment tasks. CADAL Quiz was introduced to provide a structure for tutors and lecturers to work as a team and share in the responsibility of developing assessment questions.

2. To encourage metacognitive learning in students. One way of malting students more aware of their learning is to provide them with self assessment questions that are tied to each week's laboratory tasks enabling them to monitor their own understanding. Unlike ordinary paper quizzes, both the student and staff can gain immediate feedback on their understanding and results. If a student is unsure of a response or result, they can discuss it with relevant staff immediately, rather than waiting for a paper quiz to be marked and returned, when the query may be less relevant.

3. Reducing the opportunity to copy or cheat in tests. As each quiz is unique, it makes it difficult for students to cheat.

4. Cutting the cost of assessment. CADAL Quiz automatically generates and corrects quizzes. This reduces staff hours required for printing, administering and marking.

5. Recording student results. A complete log is kept on who attempted the quiz, when they attempted it, specific question choices, time taken and final result. This not only serves as an assessment tool, but can be utilised in future curriculum development.

6. Flexibility for staff to govern the test. Staff decide on the number of questions per quiz, specify time slots and passwords to restrict quiz access, have the ability to view quiz logs and graph statistics.

While there are tools to develop interactive lessons on the web, such as SAMaker [Sloane and Dyreson 1996], and others [John Tasker, 1997], [Indiana University, 19971 many do not generate random sets of questions and do not record student results. The quizzes that are also online are either rigid or hard coded. These types of quizzes have limited applications and are generally useful for student self-assessment only. In overcoming the restrictions of conventional paper quizzes, a number of features were incorporated into CADAL Quiz, including:

2.1 Random ordering of questions and the A, B, C, D choices.

Quizzes are generated from a database of questions. This involves randomly selecting a specified number of questions from the database, and randomly ordering the chosen questions. The multiple choice alternatives (A, B, C, D) are also randomly ordered, so that if two students have the same question, the questions will appear to be different, which minimises cheating,

2.2 Results handling and analysis features

a) Immediate assessment with logged details.

The fact that each quiz is corrected immediately is a major advantage over paper quizzes. At the same time, student results are logged and can be analysed immediately. Logged information includes the student's name, ID number, email address, demonstrator's email address, the quiz attempted, the time it was attempted, the time taken to complete the quiz, the student's response to each question and the result.

b) Results optionally displayed to the student.

During self-assessment, it is acceptable for the student to see their results immediately. However, in the case of a test, results can be hidden from the student until after all students have attempted the test, which again aims to minimise cheating. The same applies to emailing students their results for future reference.

c)Results optionally emailed to staff/supervisors.

In the case when the quiz is used as part of laboratory preparation, it is convenient to have the student s results emailed to the lab demonstrator for recording. This also applies if the quiz is used as a survey. This can be turned off if not required.

Logged information can all be viewed online. In the case of student responses to questions and results, these can be graphed online, to indicate the more difficult/easier questions. This helps locate student strengths and weaknesses. The fact that this can all be done straight after or during the administration of a quiz, means that a difficult topic can be revised as soon as it becomes a problem.

2.3 The ability to subdue the randomness and specify a question breakdown.

To ensure that certain questions or topic areas are included in the quiz a question breakdown can be specified. Questions can be chosen from a range of questions, for example, select 3 questions from questions I to 10 or include question 24. This then guarantees that ranges of questions are selected, perhaps ensuring that several harder questions are included.

2.4 Administration features (such as adding and viewing questions online).

To make it easier for staff to insert questions into the quiz databases, questions can be added online. This includes password restricted access and step by step instructions to adding a new question. Staff can also view all questions in the database online, as opposed to a student who only sees a random portion.

2.5 Restricted access to quizzes and secure staff areas.

Most quizzes are available at any time, but in the case of tests it might be necessary to restrict access to certain people at various times. For example a test can be conducted over the course of a week, and only certain people can access it at any one time using time specific passwords. Access to staff areas is also password restricted.

3. Integrating CADAL Quiz into First Year Computer Programming

In the past it has been common practice to assess First Year Computer Science students through laboratory exercises, a multiple choice mid-semester test and an exam. The laboratory exercises are marked out of 10; 3 marks towards preparation and 7 marks devoted to the programming exercises. The mid-semester test was conducted during the lecture and counts towards 10% of the student's overall result for the subject [Farr and Nicholson, 19961.

With the traditional practices of assessment there has been concern about students copying preparation work and whether the mid semester test was cost-effective given the current pressure on resources and budget cuts. This year, CADAL Quiz changed the way in which students were assessed.

Each week a number of tutors devised and submitted a set of multiple choice questions into the database. These questions were related to the current week's laboratory task and aimed at testing whether the students had adequately prepared for their laboratory tasks and understood the abstractions of the lesson.

Students generated and attempted CADAL Quizzes during three practical classes. These quizzes contained 10 questions, chosen from a much larger set, and contributed to the preparation component of the student's practical mark for that week. In general the quiz took on average, approximately 10-15

minutes in most classes to administer and complete and in that time the students and tutors received details of the student's attempt. Some of the details shown in Table 1 below included: the date when the quiz was taken, the questions answered and a score out of 10. These results were automatically mailed to the demonstrator and counted towards 3 preparation marks.

=============================================== Date: Mon, 14 Apr 1997 14:09:35 +1000 (EST)

From: Online CSCIOll Quiz

Subject: Student Results - CSC IO 1 1 Quiz

Reply-To: usemame@student.monash.edu.au

Supposedly-From: John Smith

Student: John Smith ID: 12345678

Demonstrator: tutor@cs.monaskedu.au Quiz: csciab3

Date & Time: Mon Apr 14 14:09:34 1997

Results: (3=B) (1 1=C) 4 (17=B) 21 (29) 1 (1 8=A) 24 2 ** TOTAL: 5 out of 10 **

Key: X is correct, (X=Y) is incorrect, (X) is not answered. (Where X is the question number and Y their incorrect choice.)

===============================================

Table 1: Sample email sent to staff and students

During week 5 the students were familiar with the operations of CADAL Quiz so the lecturers used CADAL to replace the traditional mid-semester test, (which was formerly run in the lecture theater over two lectures). The mid-semester test contained 50 questions chosen from approximately 165 questions and was held in the computer labs over a period of one week The randomness of the questions was subdued so that tests of comparable difficulty were generated.

4. Responses - Statistics and the Educational Impact

During its first semester of operation, the impact of incorporating CADAL Quiz was measured by gathering student performance statistics and perceptions from tutors and lecturers. From the first trial there have been beneficial effects for the tutors and course lecturers as wen as students.

4.1 Tutor and Demonstrator Responses

Teaching staff were surveyed to provide feedback on the effect CADAL Quiz had on the operations of the laboratory classes. They were also interviewed to discuss the feasibility of designing and shaping educational assessment tasks in groups, with combined tutor and lecturer involvement.

Although the process of formulating questions and adding them to the general pool via the Web increased the sense of shared-ownership felt by the tutors there were several deficiencies in structure and execution of the quiz identified by the group. These included:

errors in the wording of submitted question,

students could make multiple submissions,

the random ordering of questions did not ensure that all quizzes were of an equal level of difficulty, difficulty in helping students when they answered a question incorrectly because it was difficult to tell which question they answered due to the random ordering

As a result of the above observations, the design of CADAL Quiz was changed prior to the mid semester test to provide focused, more personalised assistance to the students. The significant changes were:

test questions were attempted and proof read by three independent tutors for better monitoring,

students were only allowed to make one submission,

addition of the ability to subdue the random generation of questions to produce tests of equal difficulty,

releasing the total database of questions and answers, after the test was completed by all students, so that students could tell which question they answered, and the option they selected

The majority of tutors (70%) believed that the quiz was an effective way of determining whether a student had adequately prepared for the lab. Errors, both system and question design were rarely encountered.

4.2 Lecturer Review

The above changes in the structure and execution of the CADAL Quiz appear to have been very successful. Indeed feedback from lecturers was very positive under the revised framework.

CADAL Quiz automatically compiles and graphically displays the alternatives students selected for each quiz. The lecturers found this information very interesting and the online graph of the overall performance on each question very appealing. Not only has this enabled easy detection of the hardest questions (i.e. most wrong answers) but even which wrong answer is most commonly selected. As a result, lecturers have received hitherto unknowable feedback about the meanings their students are constructing.

"It really did help me pick up quickly on where the strong and weak points are.... It was very good to be able to see, at a glance, which questions they were very good at, which questions they were on average completely clueless about (4 bars of roughly similar length), which questions they had some vague idea about but were thwarted when it comes to detail(perhaps a couple of good sized bars, other small bars), and which (few) questions completely threw them" Graham Farr, CSC1011 Lecturer

The graph produced from student results determined CADAL Quiz's usefulness in steering course design. Difficult and easy questions were highlighted so that course lecturers could accurately locate the most misunderstood topics, or poorly worded questions and answers. This has allowed improvements to teaching while the course is still running.

"I will be looking in future lectures to further emphasise some of the many points where they are weak.. Graham Farr, CSC I OI I Lecturer

4.3 Student Results

CADAL Quiz was particularly useful m making students more aware of their own learning. In particular students decided whether they needed to do one or more of the randomly generated quizzes. These thought processes are all associated with enhanced metacognition.

With respect to the mid-semester test, a total of 365 students completed the test of 50 questions, with an average result of 60% (Standard Deviation 7.55, Range 7 - 49), which is comparable to the 1996 mid-semester (traditional hardcopy test) result, where 395 students sat the test and received an average mark of 60% (Standard Deviation 7.53, Range 7 - 49). With CADAL Quiz there was no indication that the students who completed the test later in the week had an advantage. The percentage of correct answers for each question varied from 98.5% (1 incorrect response out of 67) indicating a particularly easy question, right down to 10% correct and below, indicating particularly hard or poorly worded questions.

5. Conclusion and Future directions

CADAL Quiz is an application of WWW technology that has had a significant impact on educational assessment materials on the Internet. The process of formulating questions has increased the sense of shared ownership felt by tutors for the course. The lecturers have received hitherto unknowable feedback (from the aggregate statistics) about students understanding and the meanings their students are constructing. This has allowed lecturers to adjust their teaching while the course is still running.

The quizzes were particularly useful in testing the students understanding, The personalised feedback made the students more aware of their own learning, hence enhancing their metacognitive skills.

CADAL Quiz has several advantages over paper based quizzes. These include ease of automatic marking, ease of creation of individualised tests, and immediate feedback to students. Continued changes and improvements will make CADAL Quiz one of the most functional Web-based testing methods available.

6. Acknowledgments

The author wishes to thank the first year Computer Science lecturers Dr Graham Farr, Dr Ann Nicholson, for using CADAL Quiz in their course and all the tutors for their dedicated efforts in devising questions that undoubtedly contributed to the success of CADAL Quiz.

7. References

[Godfrey, 1996] Godfrey, R. (1996). The Word Wide Web: A replacement, displacement, supplement or adjunct of traditional methods?, Thirteenth Annual Conference of the Australian Society for Computers In Learning in Tertiary Education, Australia, pp 221-234

[Conway, 1993] Conway, D. (1993). Improving Educational Outcomes with Computer-Mediated Presentation, International Conference on Computers in Education (ICCE'93), Taiwan, pp.325-327

[Conway, 1994] Conway, D. (1994). Student Responses to Hypermedia In Lecture Theatre: A case study. World Conference on Educational Hypermedia (ED-MEDIA'94), Canada, pp 141-146

[Carbone, 1997] Carbone, A. (1997). Retaining the Best Students - A First Year Advanced Students' Project Scheme, Second Annual Australasian Conference on Computer Science Education, (ACSE'97), Australia, pp 141-148

[Carbone, 19971 Carbone, A. (1996). The Evolution of a First Year Advanced Students' Project Scheme, Technical Report 971314, Department of Computer Science, Monash University, Australia.

[Sloane and Dyreson, 1996] Sloane, A.M. & Dyreson, C. (1996). An Interactive Self-Paced Learning Environment for the World Wide Web, First Australasian Conference on Computer Science Education, Australia, pp.344-351

[John Tasker, 1997] John Taskers Quiz Quest (1997). http://dialspacp- dial.pipex.com/town/plaza/fg5O/

[Indiana University, 1997] http://www.bestindiana.edu/quizsiteVquickstart.html (1997). QuizSite Quick Start Bureau of Evaluative Studies & Testing, Indiana University

[Farr and Nicholson, 1997] Nicholson, A. & Farr, G. (1997). CSC1011 Computer programming, Lecture Notes, Monash University, Australia

This paper first appeared in Proceedings of WebNet 97 - World Conference of the WWW, Internet & Intranet - Toronto, Canada; November 1-5, 1997[CD-Rom edition], Association for the Advancement of Computing in Education, Charlottesville, VA. It is republished with permission of AACE.